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ABSTRACT 

The variables of practice, incentive, instruction, 
and feedback vere examined to determine which contributed 
significantly to the effectiveness of an instructional program, which 
was systematically developed to include the four variables. Two 
studies were carried out using the Aircraft Instrument Comprehension 
Program and Test, In the first study the effects of practice and 
incentive on learner posttest performance were examined, using two 
levels of each variable in a 2 x 2 factorial design. A second study 
examined the effects of instruction and feedback using the same 
design. Results indicated that instruction had a strong effect on 
learner performance in that scores on the posttest were significantly 
higher than on the pretest. Practice and incentive had a significant 
effect on the time required to perform the task. The test scores and 
statistical analyses are appended. (SA) 
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EFFECTS OF SYSTEMATIC VARIATIONS OF INSTRUCTIONAL 
VARIABLES IN A VERIFIED INSTRUCTIONAL PROGRAM 
Robert A. Reiser 
Department of Educational Technology 
Arizona State University 

The primary focus of the present research was to determine which 
variables contribute significantly to the effectiveness of an ef- 
fective instructional program. The variables examined in the present 
series of studies were practice, inc<>ntive, instruction., and feed- 
back. While nany studies have examined the effects of these variables, 
few studies have used an effective instructional program as the 
research vehicle. The effect of variables in instructional materials or 
other prose passages that have not been demonstrated to be effective 
may differ from the effects of variables in effective instructional 
programs. 

The instructional program used in this study was the Aircraft 
Instrument Comprehension Program (Higgins, 1973). The Aircraft Instrument 
Comprehension Program is a self-instructional program designed to train 
learners to perform an aircraft instrument comprehension task: to identify 
which one of four pictures of an aircraft in flight most nearly corresponds 
to the position indicated on an attitude indicator and a heading indicator. 
Tryouts of the program indicated that it usually enabled learners to 
perform the task at criterion-level prof ciency (Higgins & Keams, 1973; 
Higgins, Keams, & Tenpas, 1974). 
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The Aircraft Instrument Comprehension Program was systematically 
developed to Include Instruction (Instructional cues and examples), 
practice, feedback, and an Incentive. Two studies were designed to 
examine these variables. In the first study the effects of practice and 
Incentive on learner posttest performance were examined. Two levels of 
practice (presence and absence) and two levels of Incentive (presence 
and absence) were manipulated ^n a 2X2 factorial design. Based on the 
results of the first study, a second study was designed to examine the 
e^'fects of Instruction and feecback on learner posttest performance. 
Two levels of Instruction (presence and absence) and two levels of 
feedback (presence and absence) were manipulated in a 2X2 factorial 
design. In both studies learner posttest performance was reflected by 
two measures: number of correct responses and time required to complete 
the posttest. 

Experiment 1: Effects of Practice and Incentive 

Method 

Subjects . Forty-eight students enrolled In the Air Force Reserve 
Officer Training program at Arizona State University served as subjects 
In this study. 

Materl als . The materials used In this study were variations of 
the Aircraft Instrument Comprehension Program (Higglns, 1973). The program 
was designed to enable students to achieve the following Instructional 
objective: 

Given four Illustrations of aircraft In roll, pitch, 
and heading, the student will Identify the Illustration 
that most nearly represents the position Indicated on 
a compass and an artificial horizon. 
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Instruction in the program consisted of one instructional cue and three 
examples for each of the three concepts presented: roll, pitch, and 
heading. Eight examples were also presented in which these concepts were 
contined. Practice in the program consisted of one to four practice items 
for each concept, followed by ten additional practice items at the end 
of the program. Ali practice items required students to Identify which 
of two or more drawings of an aircraft in flight most nearly represented 
the position shown on an attitude indicator and a heading indicator. Upon 
completion of each practice item students were able to obtain feedback^ 
in the form of knowledge of correct response, by moving a paper slide. 

Procedure. A pretest was administered to all subjects prior 
to initiating the experiment. The pretest consisted of nine items 
selected to be representative of the items on the posttest. The 
differences in pretest scores between treatment groups w^re not 
statistically significant. 

The four treatment groups which resulted from the 2X2 factorial 
design w?re practice and incentive, practice and no incentive, no practice 
and incentive, and no practice and no incentive. Subjects attended classes 
in four different sections. In order to prevent subjects in the no 
incentive groups from being aware of the incentive, all subjects in the 
first two sections were assigned to the no incentive groups. All 
subjects were randomly assigned to either the practice or no practice 
groups. 

At the start of the experimental session each subject received 
an Instructional booklet. When all subjects had received their booklets, 
they were instructed to begin reading the booklet that had been given 
to them* 
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Subjects in the incentive groups received written instructions 
located at the beginning of their instructional booklets indicating that 
they could improve their course grade by up to one-ha'f of a grade point 
by responding correctly to the itens on the posttest. In addition, the 
instructions indicated that the four subjects who responded correctly to 
the largest number of po-ttest items in the least amount of time would 
have an opportunity to fly a formation trainer at Williams Air Force Base. 
Reserve Officer Training candidates at Arizona State University do not 
get many chances to fly a formation trainer, and value any opportunity 
to do so. Instructions to the subjects in the no incentive groups simply 
stated that the subjects' grades in the course would not be affected by 
their participation in the experiment and that the developers of the self- 
instructional program would appreciate their best efforts. 

The booklets for subjects in the practice groups contained Jll 
the instructional cues, examples, and practice items that originally 
appeared in the program. Subjects in the no practice groups received 
instructional bookets in which all practice items were deleted from the 
program. Booklets for each subject in these groups contained only the 
instructional cues and examples for each conept taught in the progr?jn. 

The last page of each booket instructed each subject to record the 
time he had finished his booklet and to raise his hand so that a proctor 
could collect his materials. A proctor checked the completion time the 
subject had listed and collected the instructional materials. The posttest 
was adninistered after all subjects had completed the instructional book- 
lets. Subjects were again asked to record their completion times, and 
the test bookets were collected as each subject finished. 
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Criterion Measur e- The Aircraft Instrunent Comprehension Test: 
Form B (Keams, Tenpas, & Higglns, 1973) served as the posttest. The 
test contained directions, a sample test Item, and 36 multiple- choice 
Items. All test items required subjects to Identify which one of four 
drawings of an aircraft In flight most nearly represented the position 
shown on an attitude Indicator and a heading Indicator. 

Data Analyses . Analyses of variance were performed to determine 
the individual and ccmblnod fjffects of practice and Incentive on post- 
test scores and posttest Items? 
Results 

The mean posttest scores of the treatment groups (See Table 1) 
were all within one point of the overall mean score of 33.31. £-rat1o$ 
for Incentive, practice, and the practice X incentive Interaction were 
not statistically significant. 



Insert Table 1 about here 



Subjects spent differential amounts of time answering items on 
the posttest. flean posttest times by treatment are reported in Table 2. 
Posttest time differences were analyzed using an .analysis of variance, 
as shown in Table 3. Statistically significant differences were found 
for both practice, {£.{1, 44) = 22.3, £< .001) and for incentive 
(F (1, 44) = 15.4, £< .001). The practice X incentive interaction 
was not significant. 
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Insert Table 2 and Table 3 about here 



In order to determine whether instruction had an appreciable effect 
on learner performance, subject pretest performance was compared with 
subject posttest performance. On the pretest, subjects answered a 
mean of 69.4% of the items correctly. On the posttest, subjects answered 
a mean of 92. 5X of the items correctly. These percentages were approx- 
imately the same for all four treatment groups. Thus, it appears that 
the improvement in subject scores from pretest to posttest was attributable 
to instruction, the one variable received by all four treatment groups. 

Experiment 2; Effects of Instruction and Feedback 

Method 

Subjects . Si xti»- three students enrolled in the Air Force Reserve 
Officer Training program at the University of Arizona served as subjects 
in this study. 

Materials. The materials used in this study were similar to those 
used in Experiment 1. The only difference in materials was that in 
this study subjects were able to obtain feedback, in the form of know- 
ledge of correct response, by marking a chemically treated answer sheet. 

Procedures . The four treatment groups which resulted from the 2X2 
factorial design were instruction and feedback, instruction and no feed- 
back, no instruction and feedback, and no instruction and no feedback. 
Performance incentives, contingent upon both the speed and accuracy of 
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a subject's posttest responses, were Identical to those described In 
Experiment 1. Subjects were Informed of the Incentives through oral 
directions given at the beginning of the experimental session. The 
Incentives were Identical for all groups « 

At the beginning of the experimental session a pretest was administered 
to the subjects. The pretest was the same pretest that was used In 
Experiment 1. Subjects who received a score of seven or more on the pre- 
test were considered to be proficient on the task prior to the studty and 
were eliminated from all the data analyses reported. The differences 
In pretest scores between treatment groups were not statistically 
significant. 

Following completion of the pretest, all subjects were given an 
instructional booklet, a marking pen, and a response sheet. Subjects 
In the Instruction groups received a booklet with all Instructional 
cues, examples, and practice Items Intact. Subjects In the no Instruc- 
tion groups received a booklet containing all the practice Items, however, 
all Instructional cues and examples were deleted. Subjects In the feedback 
groups received a special marking pen and a chemically treatoH ro^pon^e 
sheet which enabled the subjects to receive feedback In the form of 
knowledge of correct response. Subjects In the no feedback groups 
received a marking pen and a response sheet which did not provide any 
feedback. 

When all subjects had received their materials, they were Instruc- 
ted to begin reading the booklet that had been given to them. The last 
page of each booklet Instructed each subject to record the time he had 
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finished his booklet and to raise his hand so that a proctor could collect 
his materials. A proctor checked the coinpletlon time the subject had 
listed, collected the materials, and gave the subject a copy of the post- 
test. The time the subject began the posttest was recorded by the proctor. 
When a subject finished the posttest he recorded his completion time, had 
his material collected by a proctor, and was dismissed. 

Criterion Measure . The criterion measure used In this study was 
identical to the criterion measure described In Experiment 1. 

Data Analyses . Analyses of variance were performed to determine 
the Individual and combined effects of Instruction and feedback on 
posttest scores and posttest times. 
Results 

The mean score for each group on the 36-1tem posttest Is shown in 
Table 4. The mean score for the Instruction groups was 4.22 points 
higher than the mean score for the no Instruction groups; a difference 
which Is significant at the .05 level (F (1. 37) » 4.73). F-ratios for 
feedbacis and the instruction X feedback interaction were not significant 
(see Table 5). 



Insert Table 4 and Table 5 about here 



Mean times required for each group to complete the posttest are 
reported in Table 6. The mean time for the instruction groups was 1.07 
minutes less than the mean time for the no instruction groups; a 
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difference which Is significant at the .05 level (F (1. 37) ■ 4.45). 
No significant differences for either feedback or the Instruction X 
feedback interaction were observed (see Table 7). 



Insert Table 6 and Table / about here 



Discussion 



The primary focus of the present research was to detemine which 
variables contribute significantly to the effectiveness of an effective 
instructional program. The results of the two experiments indicate that 
instruction had a strong effect on learner performance on the aircraft 
instnment convrehension task. In ExpeHment I instruction appeared to 
be responsible for an impressive improvement in subject scores from pretest 
to posttest. The effect of instruction was so powerful that there was 
little room for either practice or incentive to cortrtbute significantly 
to subject posttest scores. In Experiment II instruction had a sIgnlTlcwt 
effect on posttest scores and posttest time. 

The results of Experiment I indicate that practice and incentive 
had a significant effect on the time required to perform the aircraft 
instrument comprehension task. Time is an improtant ftspec^ of the task 
in that it is an indication of the rate at which pilots read their conirol 
instruments. The ability to read instrunents accurately and quickly is 
a highly desirable skill to acquire in pilot training. Thus, practice 
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and Incentive were valuable because they Increased the speed at Mhlch 
subjects Mere able to perform the task. 

The present research Indicates that Instruction was the most 
po»*erfu1 variable In an effective Instructional program. Practice, 
Incentive, and feedback, variables that have been effective In other 
Instructional settings, did not have a significant effect on subject 
posttest scores In this study. These results may not hold across tasks 
of different degrees of complexity. In effective instructional programs 
that teach more complex tasks. Instruction alone m^y not enable learners 
to perform the tasks at criterion-level. Other variables, such as 
practice, feedback, and Incentive, m^y have to be Included In the 
programs In order for the progrwis to be effective. Further research 
should be conducted to determine the general liability consistency 
of the present findings. 
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TA8LE 1 

Mean Posttest Scores by Treatment 
Experiment I 



Incentive 


Tr~ 

Practi ce 


Practice 

No Practice 


Totals 


Incenti ve 


33.33 


32.42 


32.88 


No Incentive 


33.50 


34.00 


33.75 


Totals 


33.42 


33.21 


33.31 



n = 12 per cell 
Maximum score - 36 
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TABLE 2 

Hean Posttest Times (Minutes) by Treatment 
Experiment 1 



Practice 

Incentive 

Practice No Practice Totals 

Incentive S.98 8.13 7.05 

No Incentive 7^79 9J0 8.70 

Totals 6.88 8.86 7.88 



n « 12 per cell 



TABLE 3 

Analysis of Variance: Posttest Tines 
Experiment 1 





Source of Variation 


SS 


df 


m 


F-Ratio 



Incentive 


14.40 


1 


14.40 


15.42* 


Practice 


16.56 


1 


16.56 


22.31* 


Incentive X Practice 


1.54 


1 


1.54 


0.04 NS 


Within 


28.70 


44 


.65 





* £ < .001 

NS • Not Significant 
o 15 
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TABLE 4 

Mean Posttest Scores by Treatment 
Experiment 2 





Instruction 


Feedback 

Feedback 


No Feedback 


Totals 


Instruction 


32.32 


31.73 


32.06 


No Instruction 


31.67 


25.24 


27.84 


Total s 


31.99 


28.48 


29.95 



n » 11 per cell 
Maximum score 36 



TABLE 5 

Analysis of Variance: Posttest Scores 
Experiment 2 



Source of Variation 


SS 


it 




F-Ratio 




Instruction 


18.05 


1 


18.05 


4.73* 




Feedback 


8.69 


1 


8.69 


2.28 


NS 


Instruction X Feedback 


5.07 


1 


5.07 


1.33 


NS 


Within 


140.99 


37 


3.81 







* £ < .05 

NS = Not Significant 
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TABLE 6 

Mean Posttest Times (Minutes) by Treatment 
Experiment 2 



Feedback 

Instruction 

Feedback No Feedback Totals 

Instruction 7.18 7.11 7.14 

No Instruction 8.68 7.50 8.21 

Totals 7.93 7.30 7.61 

n » 11 per cell 



TABLE 7 

Analysis of Variance: Posttest Times 
Experiment 2 



Source of Variation 


SS 


df 




F-Ratio 


Instruction 


.89 


1 


.89 


4.45* 


Feedback 


.39 


1 


.39 


1.95 NS 


instruction X Feedback 


.30 


1 


.30 


1.50 NS 


Within 


7.75 


37 


.20 





* £ < .05 

NS « Not Significant 
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